The subject matter discussed in the background section should not be assumed to be prior art merely as a result of its mention in the background section. Similarly, a problem mentioned in the background section or associated with the subject matter of the background section should not be assumed to have been previously recognized in the prior art. The subject matter in the background section merely represents different approaches, which in and of themselves may also correspond to implementations of the claimed inventions.
The last few decades have witnessed a fundamental change in the challenge of taking in new information. The bottleneck is no longer access to information; now it is our ability to keep up. We all have to read more and more to keep up-to-date with our jobs, the news, and social media. AI can improve people's work by helping with this information deluge. One answer is to use a deep reinforced model for abstractive summarization to automatically summarize longer texts.
Automatic text summarization is a field of natural language processing that is increasingly used in industry today. The goal of the summarization process is to create a summary of one document or multiple documents that will retain the sense and the most important aspects while reducing the length substantially, to a size that may be user-defined. Training a model that can generate long, coherent, and meaningful summaries remains an open research problem. Generating any kind of longer text is hard for even the most advanced deep learning models.
Text summarization is the process of automatically generating natural language summaries from an input document while retaining the important points. By condensing large quantities of information into short, informative summaries, summarization can aid many downstream applications such as creating news digests, search, and report generation.
Automatic summarization models can work in one of two ways: by extraction or by abstraction. Extractive models form summaries by copying parts of the input without any modification, selecting relevant phrases of the input document, and concatenating them to form a summary. They are quite robust since they use existing natural-language phrases that are taken straight from the input, but they lack in flexibility since they cannot use novel words or connectors. They also cannot paraphrase like people sometimes do. In contrast, abstractive models generate a summary based on the actual “abstracted” content. An abstraction-based summary can compress, fuse or paraphrase sections of the source document, using words that were not in the original input, generating new phrases and possibly rephrasing. This gives a lot more potential to produce fluent and coherent summaries but it is also a much harder problem, as the model must be able to generate coherent phrases and connectors.
Even though abstractive models are more powerful in theory, it is common for them to make mistakes in practice. Typical mistakes include incoherent, irrelevant or repeated phrases in generated summaries, especially when trying to create long text outputs. They historically have lacked a sense of general coherence, flow and readability.
ROUGE, which is short for recall-oriented understudy for gisting evaluation, is the name of a set of metrics and a software package usable for evaluating automatic summarization in natural language processing. ROUGE works by comparing matching sub-phrases in generated summaries against sub-phrases in ground truth reference summaries, even if they are not perfectly aligned. That is, the metrics compare an automatically produced summary against a set of human-produced summaries.
Recent neural network models based on the attentional encoder-decoder model for machine translation (Nallapati et al., 2016; Zeng et al., 2016) have been able to generate abstractive summaries with high ROUGE scores. However, these systems have typically focused on summarizing short input sequences of one or two sentences, to generate even shorter summaries—for example with a limit of 75 characters.
Nallapati et al. (2016) applied their abstractive summarization model on the CNN/Daily Mail dataset (Hermann et al., 2015), which contains input sequences of up to 800 tokens and multi-sentence summaries of up to 100 tokens. The analyses by Nallapati et al. (2016) illustrate a key problem with attentional encoder-decoder models: they often generate unnatural summaries consisting of repeated phrases.
The disclosed robust and coherent abstractive text summarization model addresses these issues of general coherence, flow and readability, as well as unnatural summaries with repeated phrases. Other aspects and advantages of the technology disclosed can be seen on review of the drawings, the detailed description and the claims, which follow.